Virtual Hot Inserting Functions in a Shared I/O Environment

ABSTRACT

A blade server system and method for virtually hot plugging and virtually hot removing functions in a shared I/O environment. A management node physically hot inserts and hot removes an I/O node in the server system without a compute node being aware of the hot insert and hot removal. The management node and the compute node create and remove virtual links between the compute node and the virtual functions.

BACKGROUND

Blade servers are self-contained all inclusive computer servers,designed for high density. Blade servers have many components removedfor space, power and other considerations while still having all thefunctional components to be considered a computer (i.e., memory,processor, storage).

The blade servers are housed in a blade enclosure. The enclosure canhold multiple blade servers and perform many of the non-core services(i.e., power, cooling, I/O, networking) found in most computers. Bylocating these services in one place and sharing them amongst the bladeservers, the overall component utilization is more efficient.

In a non-shared I/O environment, there is a direct physical link betweena host computer on a compute node and an I/O node. The functions of theI/O node are typically assigned to a single host. Thus, when an I/O nodeis inserted into the server system, one of the hosts can request itsfunctions and the management module for the server enclosure assigns thefunctions of that I/O node to the requesting host. In other embodiments,the assignments of I/O functions are made implicitly by blade and I/Onode slot mappings designated by the enclosure manufacturer.

These types of enclosures are not very efficient because other hostscannot utilize I/O node functions until the host presently using the I/Ofunction gives up control. Additionally, there might be functions on aparticular I/O node that are not being used by the assigned host butcould be used by other hosts in the system.

Virtualization of functions in a shared I/O environment enablesfunctions of an I/O node to be shared across many hosts. Once thefunction is assigned to a host, the host thinks that it owns the I/Ofunction. However, adding and removing functions from the host'sassigned functions means executing the normal physical steps for addingand removing I/O nodes.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 depicts a block diagram of one embodiment of a server system.

FIG. 2 depicts a host view of virtual functions assigned to the host.

FIG. 3 depicts a flow chart of one embodiment of a method for physicalhot insertion of the new I/O node in accordance with the method of FIG.3.

FIG. 4 depicts a flow chart of one embodiment of a method for virtualhot insertion of a virtual I/O function.

FIG. 5 depicts a flow chart of one embodiment of a method for virtualhot removal of a virtual I/O function.

FIG. 6 depicts a flow chart of one embodiment of a method for physicalhot removal of an I/O node from the server system.

DETAILED DESCRIPTION

The following detailed description is not to be taken in a limitingsense. Other embodiments may be utilized and changes may be made withoutdeparting from the scope of the present disclosure.

FIG. 1 illustrates a block diagram of one embodiment of a server systemthat can incorporate the virtual hot plugging functions of the presentembodiments. The illustrated embodiment has been simplified to betterillustrate the operation of the virtual hot plugging functions.Alternate embodiments may use other functional blocks in which thevirtual hot plugging functions can operate.

The system is comprised of a plurality of compute nodes 101-103. In oneembodiment, the compute nodes 101-103 can be blade servers. The serversmay be comprised of components that include a processor, memory, and I/Ointerfaces (e.g., PCI Express).

The system is further comprised of I/O nodes 110-112. The I/O nodes110-112 can be typical I/O devices that are used in a computer serversystem. The I/O nodes 110-112 are each comprised of one or more virtualfunctions 140-142. Such I/O functions can include serial and parallelI/O, fiber I/O, switches (e.g., Ethernet switches), and other functions.

The I/O nodes 110-112 are coupled to the compute nodes 101-103 through aswitch platform 121. Any one of the I/O nodes 110-112 can be switched toany one of the compute nodes 101-103 through the switch platform 121.

The I/O nodes 110-112 are each coupled to the switch platform 121through a physical hot plug connection 150. This connection 150, in oneembodiment, is one or more connectors in the blade server system.Physical hot insertion of the I/O nodes is discussed subsequently withreference to the flow chart of FIG. 3.

Control of the switch 121 is performed by a management node 131. Themanagement node 131 is comprised of a controller and memory that enablesit to execute the control routines to control the switches.

The server system of FIG. 1 is for purposes of illustration only. Otherserver systems can be comprised of different quantities of computenodes, I/O nodes, and switches. The embodiments of the method forvirtual hot plugging of functions in a shared I/O environment can beexecuted in any type of server system or computer system.

Since a server system can be comprised of multiple blade servers andeach server can run multiple applications under multiple operatingsystems, such a server typically needs increased network bandwidth andmore network connections as compared to a typical computer. And sincethe server resources are shared among multiple applications, ensuringthe performance and availability of critical apps becomes moredifficult.

In traditional server environments, these issues can be resolved byresource segregation. Each server runs only one application, and each isprovided with separate I/O resources. This type of server providesmultiple physically distinct networks.

With virtualization, a flexible pool of resources can be created thatcan be deployed as needed. Any server can ideally run any application.This means that a single blade server now needs sufficient connectivityfor all of the applications it hosts. Instead of having multiple cardsand cables per server, I/O virtualization employs a single high-speedI/O link that is logically managed as multiple virtual resources.Analogous to multiple virtual machines running on a single physicalserver, virtual I/O enables the creation of multiple virtual networkinterface cards (vNICs) for network connectivity and virtual host busadaptors (vHBAs). These virtual cards behave substantially the same asthe physical Ethernet and Fiber Channel cards they are designed toreplace. Since the vNICs and vHBAs remain logically distinct, theycreate network and storage connections that also remain logicallydistinct.

The embodiments of the method for virtual hot plugging of virtualizedfunctions in a shared I/O environment, such as the system illustrated inFIG. 1, provides dynamic allocation of I/O functions across multiplehosts. The present embodiments also provide for dynamically removingthese functions without physically removing any of the I/O cards.

FIG. 2 illustrates a conceptual block diagram of one embodiment of aserver system incorporating the virtual hot plugging functions of thepresent embodiments. Such a conceptual system can be configured usingthe system illustrated in the block diagram of FIG. 1.

The conceptual system is comprised of a compute node 220 (e.g., bladeservers) with at least one host computer 200 that executes code tocontrol virtual functions. The compute node 220 has a virtual PCIExpress switch component 203 that can be coupled to the compute node'smezzanine connector and acts as a bridge between the host computer 200and a plurality of hot pluggable I/O devices 210-213.

The plurality of hot pluggable I/O devices 210-213 represent the virtualfunctions that are resident on the I/O nodes of FIG. 1. In oneembodiment, it is possible that all of the hot pluggable end devices210-213 can be resident on only one of the I/O nodes. Alternateembodiments can have the hot pluggable end devices 210-213 on differentI/O nodes.

The virtual PCI Express switch 203 is comprised of a downstreamPCI-to-PCI bridge device 225-227 for each virtual connection to avirtual hot pluggable I/O device (i.e., virtual function). A PCIPCI-to-PCI upstream interface 230 couples the host computer 200 to thevirtual PCI Express switch 203.

The hot insertion embodiments of the present disclosure encompassphysical hot insertion of an I/O node, as discussed subsequently in FIG.3, and virtual hot insertion of a virtual function from the I/O node, asdiscussed subsequently in FIG. 4. The embodiments also encompass virtualhot removal of the virtual function from the I/O node, as discussedsubsequently in FIG. 5, and physical hot removal of the I/O node, asdiscussed subsequently in FIG. 6.

FIG. 3 illustrates a flow chart of one embodiment of a method forphysical hot insertion of a new I/O node in a server system. In oneembodiment, this method can be handled by the management node with thecompute node uninvolved and unaware of the insertion event.

The new I/O node hardware is inserted into the system 301. Themanagement node is then notified that a hot insertion is being requested302. This notification can take the form of the user pushing anattention button to send an interrupt to the management node indicatingthat a new card has been inserted into a particular slot of the chassis.For example, the user might interface with a keyboard and monitor or atouch screen input device that allows selection of the type of card andthe slot into which the card has been inserted. In another embodiment,circuitry on the inserted card automatically sends a signal to themanagement node that it has been inserted. The inserted card can alsoinclude embedded information interrogated by the management node toascertain the various functions available on the new hardware.

The management node provides a visual indication that the hot plugoperation is in progress 303. This can be accomplished by the managementnode setting indicator control bits on the switch platform. Themanagement node then turns on power to the I/O slot 304. In oneembodiment, this is accomplished by the management node setting powercontrol bits on the switch platform.

The management node provides a visual indication that the hot plugoperation is complete 305. The management node can perform this bysetting the power indicator control bits on the switch platform.

At this point, the compute node is still unaware that an I/O resourcehas been added. The management node can then control the connections ofthe virtual functions to the compute node as discussed subsequently withreference to FIG. 4. Once connected, each virtual function will appearto be a single function I/O device to the compute node.

The management node connects virtual functions to the compute nodesbased on resource requests that can be provided through a consoleconnected to the management node. FIG. 4 illustrates a flow chart of oneembodiment of a method for virtual hot insertion of a virtual function.This embodiment can also be referred to as a hot add event wherein poweris virtually added to a virtual function on an existing I/O device. Thisembodiment involves both the management node and the compute node. Themanagement node controls bits in the slot control register of theappropriate PCI-to-PCI downstream configuration spaces as necessary tomake the virtual hot insertion of one of the virtual functions to appearto be an actual physical hot insertion.

The host computer is notified that a hot insertion operation is beingrequested 401. This can be accomplished by the management node settingthe attention button pressed bit in the slot status register in thedownstream PCI-to-PCI bridge device. This is equivalent to a virtual“push” of the attention switch.

The compute node was initially unaware of the physical hot insertion ofthe I/O node. Only after the management node has changed the appropriatecontrol bits does the compute node recognize and believe that a physicaldevice has now been hot inserted. The compute node will then perform aPCI enumeration that discovers, initializes, and enables the newly-addedvirtual functions.

The host computer provides a visual indication that the hot insertionoperation is in progress 403. This may be accomplished by the hostcomputer setting power indicator control bits in the slot controlregister in the downstream PCI-to-PCI bridge device. The management nodeis interrupted when these bits are set.

The host computer then turns on power to the I/O slot 405 by using thepower control bits in the slot control register in the downstreamPCI-to-PCI bridge device. The management node is interrupted when thesebits are set. The host computer believes that it is requesting thatpower be applied to the physical I/O node but in reality, the managementnode has done this at a much earlier time.

In response to the “power on” request from the host computer, themanagement node makes the host computer believe that the physical I/Onode has been powered up and the PCI Express link is now active 407. Inone embodiment, this is done by the management node setting the datalink layer active bit in a link status register and the data link layerstate changed bit in the slot status register in the downstreamPCI-to-PCI bridge device.

The host computer provides a visual indication that the hot insertionoperation is complete 409 by setting the power indicator control bits inthe slot control register in the downstream PCI-to-PCI bridge device tothe “on” state. The management node is interrupted when these bits areset and is thus informed that the virtual hot insertion operation iscomplete.

FIG. 5 illustrates a flow chart of one embodiment of a method forvirtual hot removal of a virtual function. This method involves both themanagement node and the compute node. The management node controls bitsin the slot control register of the appropriate downstream PCI-to-PCIbridge configuration spaces as necessary to make the virtual hot removalof one of the virtual functions appear to be the physical hot removal ofthe physical I/O node.

The management node removes the virtual functions from the compute nodesbased on the resource requests that can be provided through a consoleconnected to the management node. Since the compute node believes that aphysical single-function device has been hot removed, it will quiesce,disable and power down the function.

The host computer is informed of the hot removal operation request 501.This can be accomplished by the management node setting the attentionbutton pressed bit in the slot status register. This provides a virtual“push” of the attention switch.

The host computer provides visual indication that the hot removaloperation is in progress 502. This is accomplished by the host computersetting power indicator control bits in the slot control register to the“blink” state. The management node is interrupted when these bits areset.

The host computer then quiesces traffic to and from the indicated I/Odevice 504. The host computer turns off power and the power indicator ofthe slot 505 by setting the setting power control bits in the slotcontrol register. The management node is interrupted when these bits areset. The host computer believes that this particular physical I/O deviceis being powered down.

In response to the power off request, management node makes the hostcomputer believe that the physical I/O node has been powered down 507.This can be accomplished by the management node clearing the data linklayer active bit in the link status register and setting the data linklayer state changed bit in the slot status register.

The host computer then provides a visual indication that the hot removaloperation is complete 509 by setting the power indicator control bits inthe slot control register to the “off” state. The management node isinterrupted when these bits are set and now knows that the virtual hotremoval operation is complete.

FIG. 6 illustrates a flow chart of one embodiment of a method forrequesting physical removal of an I/O device from a server system. Sincethe compute node believes that the physical I/O node has already beenremoved, this method is performed by the management node while thecompute node is unaware of the event. The connections between the hostcomputers and the virtual functions in the I/O node are virtuallyremoved before physical hot removal of the I/O node.

A request is received to remove the physical I/O device 601. This can beaccomplished by a user pressing an attention button assigned to thephysical slot of the I/O device. The I/O node signals the managementnode of the request to remove the physical I/O device. In oneembodiment, the I/O node sends an interrupt to the management node torequest removal.

The management node provides a visual indication that the hot removaloperation is in progress 607. This is accomplished by the managementnode setting power indicator bits to the “blink” state on the switchplatform.

The management node then verifies that virtual functions are notvirtually connected (in service) to the host computers 603. If the I/Onode still has active connections to some of the host computers, themanagement node virtually hot removes all of the connected virtualfunctions 605.

Once it has verified that all virtual functions are removed 603, themanagement node then turns off the power to the I/O node slot 609. Themanagement node can accomplish this by using the power control bits onthe switch platform.

The management node then provides a visual indication that the hotremoval operation is complete 611 by setting the power indicator controlbits on the switch platform to the “off” state.

In summary, a method for virtually hot plugging and virtually hotremoving functions in a shared I/O environment has been disclosed. Whensharing I/O functions across multiple hosts, some of the I/O functionsmay not be used. The present embodiments enable dynamic resourcing ofthese unused functions. The unused functions can also be dynamicallyremoved in a virtual manner without the physical removal of the I/Odevice from the server system.

1. A method for virtual hot inserting functions in a server system, themethod comprising: physically coupling an I/O node to the server system,the I/O node comprising a plurality of functions; and virtually sharingthe plurality of functions amongst a plurality of host computers in theserver system.
 2. The method of claim 1 wherein virtually couplingcomprises: a first host computer of the plurality of host computerssetting a virtual power control indication for a first function of theplurality of functions; setting a virtual indication that a data linklayer is active; and signaling that a virtual link between the firsthost computer and the first function is active.
 3. The method of claim 2wherein setting the virtual indication that the data link layer isactive comprises a management node setting a data link layer active bitin a link status register.
 4. The method of claim 3 wherein themanagement node setting the data link layer active status bit comprisessetting the data link layer active status bit in response to the firsthost computer setting the virtual power control indication.
 5. Themethod of claim 1 wherein virtually coupling comprises setting up avirtual connection between a first host computer of the plurality ofhost computers and a first function of the plurality of functions. 6.The method of claim 2 and further comprising the first host computerenumerating virtual functions assigned to the first host computer. 7.The method of claim 2 and wherein signaling that the virtual linkbetween the first host computer and the first function is active thatincludes providing a visual indication of completion of the virtual hotinsertion.
 8. A method for virtual hot removal of functions in a serversystem, the method comprising: virtually indicating a hot removalrequest of a first function of a plurality of functions on an I/O node;and virtually removing a virtual link of the first function to a firsthost computer.
 9. The method of claim 8 and further including physicallyremoving the I/O node from the server system after removing the virtuallink.
 10. The method of claim 8 wherein virtually removing the virtuallink comprises: the first host computer quiescing traffic to the firstfunction; the first host computer setting power control bits to turn offpower to the function; and a management node clearing a data link layeractive bit in a link status register.
 11. The method of claim 8 whereinvirtually indicating the hot removal request of the first functioncomprises a management node setting an attention button pressed bit inan I/O node status register.
 12. The method of claim 10 and furtherincluding the first host computer providing a visual indication ofcompletion of the virtual hot removal of the first function.
 13. Aserver system incorporating a plurality of virtual functions, the systemcomprising: a plurality of compute nodes each configured to virtuallycouple to at least one virtual function of the plurality of virtualfunctions; an I/O node comprising the plurality of virtual functions; aswitch platform that couples the plurality of host computers to the I/Onode; and a management module coupled to the switch platform andconfigured to virtually hot insert and virtually hot remove virtuallinks between each of the plurality of compute nodes and the at leastone virtual function.
 14. The system of claim 13 wherein only themanagement node is further configured to physically hot insert andphysically hot remove the I/O node to and from the server system. 15.The system of claim 11 wherein both a first compute node and themanagement node are configured to virtually hot insert and virtually hotremove the virtual link between the first compute node and the at leastone virtual function.